1. Field of the Invention
The present invention relates to reduction of motion compensation artifacts in a high resolution image interpolation and, more specifically, to halo reduction in a hierarchical approach.
2. Discussion of Related Art
Image interpolation based on motion compensation is a well-established technique for frame rate conversion (FRC) and is often utilized to increase the refresh rate in video. In such applications, motion appears more fluid and a high refresh rate is more suitable, for example, for LCD panels.
FRC is very challenging to perform, however, particularly in high definition television HDTV. Compared to standard definition video, HDTV involves larger picture formats and also larger motion vectors MV in terms of absolute pixel numbers. These differences in HDTV result in expensive motion estimation and in a large halo region.
Expensive motion estimation ME involves using suitable approaches for complexity reduction that may include, for example, a 3-step search, a log D step search, and a hierarchical approach. Hierarchical-search approaches, which are more generic than the 3-step or logD-step searches approaches, utilize appropriate filtering for each image reduction.
Halo reduction, even for standard definition television SDTV, is still an active area of research. R. Thoma & M. Bierling, “Motion Compensating Interpolation Considering Covered and Uncovered Background”, Signal Processing: Image Communication 1, 1989, pp 191-212, describes a system that involves both hierarchical search and halo reduction. The authors suggested a hierarchical approach for ME complexity reduction and halo detection for the limited case of still backgrounds, which can be utilized in the teleconferencing environment. Other solutions have been suggested that attempt to address independently one of the hierarchical ME approach or halo reduction.
Not restricted only for HDTV, the hierarchical approach for ME complexity reduction can be used for various image formats from CIF, SIF-50, SIF-60 Intermediate Formats, to SDTV-50, SDTV-60 television formats, in real-time processing. The hierarchical techniques are based on a pyramidal decomposition of an image into many sub-resolution images by appropriate anti-alias filtering and image decimation. The ME is thus evaluated from low to original (high) resolution. As previously mentioned, unfiltered versions of hierarchical approaches include the well-known three steps search, or more generally, the “logD-step” technique working directly with pixels in original resolution.
Of course, the coarse-to-fine hierarchical approaches are sub-optimal solutions to compare with the optimum exhaustive but expensive full search technique. However, a hierarchical method is commonly chosen when the image processor technology is not fast or economical enough to perform a full-resolution approach.
There have been various hierarchical algorithms developed only for ME complexity reduction. The previously cited reference from the technical paper of R. Thoma & M. Bierling, 1989, was based on the “logD-step” search technique with low-pass filtering and holes and overlapped regions correction. Similar ME using a 3-level pyramidal decomposition are presented in U.S. Pat. Nos. 5,610,658 and 5,754,237. For further complexity reduction, the disclosure in U.S. Pat. No. 6,130,912 suggested a 3-step search with integral projections on vertical and horizontal axes of each super micro-block. The disclosure in U.S. Pat. No. 6,160,850 suggests 3-step search techniques and an appropriate control unit for reducing required memory. In the HDTV application, the inventors in US 2008/0074350 A1 have suggested the use of two processor devices for horizontal sharing of the high resolution interpolation.
The most elaborated hierarchical techniques for FRC are probably described by B. W. Jeon, G. I. Lee, S. H. Lee and R. H. Park, “Coarse-to-Fine Frame Interpolation for Frame Rate Up-Conversion Using Pyramid Structure”, IEEE Transactions on Consumer Electronics, Vol. 49, No. 3, August 2003, pp 499-508. An almost identical proposal by the same two authors G. I. Lee and R. H. Park is also presented in “Hierarchical Motion-Compensated Frame Interpolation Based on the Pyramid Structure”, Y. Zhuang et al. (Eds.): PCM 2006, LNCS 4261, pp. 211-220, 2006, © Springer-Verlag, Berlin Heidelberg 2006. In order to reduce holes and overlapped regions effects, since the second hierarchic level, the authors suggested an estimation of independent forward and backward MV determination. For this purpose, the authors required an interpolated image combined from the previous level estimated MV and from moving details of the two existing images. Thus, many (3) additional interpolations for each hierarchical level are performed. Moreover, in occlusion regions, estimated MV is generally not correct. Anyway, in these publications, no consideration for halo artifacts due to false MV determination is mentioned.
Holes are created in an interpolated image when there is no estimated MV from a pixel in a reference or exiting image to an interpolated image. Inversely, overlapped regions are created when there are many possible MVs from reference or exiting images to interpolated images. Halo effects are essentially due to erroneous MV estimations in occlusion areas, which occur around a foreground object in an image in a video sequence when the object is moving relative to a background.
There has been various halo reduction (HR) algorithms developed. Solutions from multi-frame (more than two frames) to two-frame solutions have been proposed. Even with better potential for HR, multi-frame solutions are expensive.
Two-frame solutions are generally composed of halo region or precisely covering/uncovering region detection and halo reduction. U.S. Pat. Nos. 6,219,436, 6,487,313 and 7,039,109 describe typical representative techniques for performing two-frame solutions. Covering/Uncovering region detection is based on some metrics such as ME error, MV length and MV border. These parameters are not necessarily reliable in an occlusion region and make the desired detection difficult. The halo reduction becomes, in turn, an ad-hoc technique using the mean or the median value provided from many possible filtering techniques.
In the cited technical publication of R. Thoma & M. Bierling, 1989, the Covering/Uncovering region detection is based on the estimated MV and the supposition of fixed background usually in teleconference applications. The halo reduction is therefore an image interpolation adaptive to detected regions. Still background supposition is somewhat specific or restrictive, and not always correct for moving television signals.
For those familiar with FRC, there are many specific cases where the ME cannot adequately provide a ‘good’ solution. Thin fast moving objects, such as a balancing hammock net, yield erroneous motion vectors in a very large region. The resulting noticeable halo, even in a still background such as a still lawn with a hammock net, is difficult to properly correct. In other cases, for example fade-in fade-out with background light turning on and off, the ME can make a foreground object disappear or re-appear.
Therefore, there is a need for better MV and ME estimations in performing FRC operations.